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10 TECHNIC AL FIELD 

This invention relates to sensing and, more particularly, to eye tracking. 

BACKGROUND OF THE INVENTION 

15 

Various eye tracking techniques are known including oculometers, such as is disclosed in 
U.S. Pat. No. 3,462,604. An example of another type of eye tracker, based on the 
detection of Purkinje images, is disclosed in U.S. Pat. No. 3,712,716. Still another 
example of a type of eye tracker is disclosed in U.S. Pat. No. 4,561,448, based on electro- 
20 oculography. These are examples only as other types of eye trackers are known. These 
can be used to track one or more axes of the attitude, i.e., the pitch, roll and yaw of the 
eyeball in its socket. Roll, i.e., eyeball torsions can be neglected and are usually not 
measured. The translatory position of the eyeball within its socket is also not measured it 
being assumed stationary with respect thereto. 

25 

Various head tracking methods are known including Polhemus Navigation Sciences U.S. 
Pat. Nos. 3,983,474 and 4,017,858 and like patents such as shown in U.S. Pat. No. 
3,917,412 to Stoutmeyer. These are used to measure the attitude, i.e., the pitch, roll and 
yaw of a pilot's head within a cockpit of a high performance aircraft. The translatory 
30 position of the head within the cockpit is not measured. It is evidently neglected and the 
center of rotation of the pilot's head is assimied to be stationary with respect to the 
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aircraft. 

It is known to combine the above described head and eye monitoring techniques as 
shown in U.S. Pat. No. 4,028,725 to Lewis. In that case, the hehnet attitude measuring 
5 system of Stoutmeyer (U.S. Pat. No. 3,917,412) is combined with an eye angle (yaw) 

detector such as shown in U.S. Pat. No. 3,724,932 to Comsweet et al. The line of sight of 
the eye angle of the observer with respect to his head plus the head angle with respect to 
the center line of the aircraft are measured to control a servoed mirror in front of the eye 
to keep it always looking at a fixed point on the display. Translatory head position is not 
10 measured with respect to any fixed coordinate system of the aircraft. 

A contact-analog headup display disclosed in U.S. Pat. No. 5,072,218 showed symbolic 
images superimposed at selected points on a pilot's visual field as the aircraft overflies 
the earth. The position and attitude of the aircraft with respect to the earth and the attitude 

15 of the helmet with respect to the aircraft are monitored in order to convert a plurality of 
stored earth position signals into helmet coordinates. Selected points on earth, such as 
flightplan waypoints, viewable through the visor of the headup display by the pilot, have 
symbolic flags planted thereon by means of the display, i.e., the waypoint symbols 
remain "stuck" on the earth, in the eyes of the pilot, regardless of the attitude of the 

20 aircraft and regardless of the attitude of the helmet. Eye attitude is not measured nor is 
there any measurement of translatory head position with respect to the aircraft. 

DISCLOSURE OF INVENTION 

25 An object of the present invention is to provide a new eye tracking method and apparatus. 

According to a first aspect of the present invention, an eye attitude monitor is combined 
with a head translatory position monitor in order to relate the eye's translatory position as 
well as its attitude to an arbitrarily selected reference coordinate system. Eye attitude can 
30 mean up to three axes of rotation (pitch, roll, yaw) about an origin of an eye coordinate 

system. The eye may be approximately assxmied to be fixed in position with respect to the 
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origin of a head coordinate system so that any translations in position of the eye with 
respect to the head may be neglected. This is a good assumption because the eye shifts its 
position very little in its socket. Its movements involve mostly "pitch" and "yaw" 
rotations. "Roll" (torsions) can be neglected as well, if desired. The assumption that the 
5 eye is "fixed" in translator^ position with respect to the origin of the head coordinate 
system makes it possible to relate the eye's translatory position to that of the head's by a 
translatory transformation of the respective coordinate systems in a simple way, i.e., 
involving constants only and not requiring any monitoring of the eye's translatory 
position with respect to the translatory position of the head. 

10 

In further accord with this first aspect of the present invention, a head attitude monitor is 
added to relate the attitude of the eye to the arbitrarily selected reference coordinate 
system. 

15 According to a second aspect of the present invention, the attitude of an eye is sensed 

with respect to an associated head coordinate system for providing an eye attitude signal, 
the attitude of the head coordinate system is sensed with respect to an arbitrarily selected 
first reference coordinate system such as a body, vehicle, or inertial reference coordinate 
system, and instead of sensing the translatory position of the head with respect to the 

20 selected first reference coordinate system it is assumed that the translatory position of the 
head is approximately fixed with respect to the selected first reference coordinate system 
and the translatory position of the selected first reference coordinate system is sensed 
with respect to an arbitrarily selected second reference coordinate system such as an 
inertial reference coordinate system; a visual axis vector signal is then provided 

25 referenced, as desired, to the selected first or second reference coordinate system for 

providing a control signal. Such may, but need not be for controlling an image according 
to the visual axis vector signal. 

The present invention provides a new way to monitor an eye, i.e., with respect to more 
30 than one coordinate system, in order to open up new opportvmities for eye-controlled 

devices including, but not limited to, image displays wherein image artifacts, nonuniform 
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image characteristics and the like may be controlled in a way heretofore not possible or 
contemplated. See for example the positioning of a nonuniform resolution spot on a 
display according to a monitored visual axis such as disclosed in copending application 
U.S. Ser. No. 08/001,736, now U.S. Patent No. 5,422,653, especially in connection with 
5 FIGS. 7(a) through 14 at page 29, line 3 through page 51, line 14 which is hereby 
incorporated by reference. 

These and other objects, features, and advantages of the present invention will become 
more apparent in light of the detailed description of a best mode embodiment thereof, as 
10 illustrated in the accompanying drawing. 

BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 shows an apparatus for eye tracking, according to the present invention, for 
1 5 providing a control signal; 

FIG. 2 shows a plurality of coordinate systems, for eye tracking, according to the present 
invention; 

20 FIG. 3 shows an edge view of a display with coordinates related to FIG. 2; 

FIG. 4 shows a method for eye tracking, according to the present invention; 

FIG. 5 shows an application of eye tracking, according to the present invention, for image 
25 control for a passive viewer; and 

FIG. 6 shows another application of eye tracking, according to the present invention, for 
image control for an active viewer. 

30 
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BEST MODE FOR CARRYING OUT THE INVENTION 



FIG. 1 shows an apparatus 10, according to the present invention, for tracking the attitude 
5 of one or both eyes 12, 14 with respect to a head 16 attached to a body 18. By "attitude" 
is meant angular rotations of an eye visual axis with respect to arbitrarily selected axes of 
an eye coordinate system. E.g., such angular rotations may include a selected pitch axis 
(e.g., supraductions above and infraductions below a level visual axis), a selected yaw 
axis (e.g., abductions and adductions, respectively, from and away from a straight ahead 
10 "primary position" with respect to the nose), or the like. 

It may be assumed that the origin of the eye coordinate system is fixed in relation to the 
origin of a head 16 coordinate system. It may therefore be related by constants, as 
discussed below. The head 16, according to the invention, is tracked at least in attitude 
15 with respect to an arbitrarily selected coordinate system such as the body, a vehicle 
within which the body is positioned, or another referent such as an inertial reference 
coordinate system. The apparatus 10 at least comprises one or more eye monitors 20, 22 
for monitoring the attitude of each monitored eye with respect to the head 16. 

20 In addition, the apparatus 10 includes a head attitude monitor 24 for monitoring the 
attitude of the head 1 6 with respect to the selected first coordinate system such as the 
body 18 or any other referent. It may also, but need not include a head translational 
position monitor 27 for monitoring the translatory position of the head 16 with respect to 
the first selected reference coordinate system or any other arbitrarily selected reference 

25 coordinate system. It may, but need not include an attitude monitor 25 for monitoring the 
attitude of the selected first coordinate system with respect to an arbitrarily selected 
reference coordinate system. Such can be a body attitude monitor 25 for monitoring the 
attitude of the body 18 or a vehicle body (within which the body 18 is positioned) with 
respect to an arbitrarily selected reference coordinate system such as, but not limited to, 

30 an inertial reference system. It may also, but need not, include a body translatory position 
monitor 26 for monitoring the translatory position of the first selected coordinate system 
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such as the body 18 (or a vehicle body within which the body 18 is positioned) with 
respect to a reference system such as an inertial reference system. The attitude and 
position monitors 25, 26 need not be separate but can combine the functions of 
monitoring the attitude of the first selected coordinate system or vehicle with respect to 
5 another reference frame such as an inertial reference frame. 

If the head attitude monitor 24 is of the type that is inherently referenced to an inertial 
reference frame then the function of the head position monitor 27 may be carried out by 
the head attitude monitor alone. In other applications it may be acceptable to assume that 
10 the head and body positions are relatively fixed with respect to each other and that the 
body position monitor 26 or the head position monitor 27 alone will suffice. 

The monitors 20, 22, 24, 27, 25, 26 provide sensed signals on lines 30, 32, 28, 35, 33, 34, 
respectively, to a computer 36 which may be a microprocessor for carrying out at least 

15 the eye-head coordinate transformations described in connection with FIG. 2 or variations 
thereof. The computer 36 provides a tracking or visual axis signal on a line 38. This may 
be provided to any control device 40 in which it may be put to good use, such as an 
image control which in turn provides an image control signal on a line 42 for controlling 
images provided by a display (not shown). Such a control signal on the line 42 has utility 

20 in positioning an image artifact, for positioning a more highly detailed portion of an 

image with respect to a lesser detailed portion, for locating a portion of an image having 
more dynamic range than other portions of the same image, for positioning an image 
artifact, for acquiring a target for controlling a projectile, or for other purposes. 

25 FIG. 2 shows four rectangular coordinate systems in an object space 44 which may be an 
inertial reference system, although other types of coordinate systems may be used as 
well. A viewer's head 50 is illustrated, similar to the head 16 of FIG. 1, having a three 
dimensional x, y, z coordinate system having an origin 52 in relation to a body 54 having 
a three dimensional x*, y*, z* coordinate system having an origin 56 and representative 

30 of a body associated with the head 50. The body may be a human body attached to the 

head or a vehicle body within which the human body is positioned. The body is shown in 
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relation to the x**, y**, z** coordinate system 44 which may be an earth reference frame 
or any object space having actual objects therein. The coordinate system 44 represents a 
ready coordinate reference system by which objects, the viewer's head 50, body and eyes, 
as well as virtual objects may be referenced to each other. It should be understood, 
5 however, that the actual measurement made by the monitors of FIG, 1 may, for example, 
measure the relation of the head axis with respect to the earth reference frame. In that 
case the body position and direction may be ignored. In such a case the body direction 
can still be monitored, e.g., in relation to the earth coordinate system, and the relation 
between the head axis and the body axis inferred, if desired, by their differences to the 

10 inertial referent. Such an approach would be usefiil where it would be undesirable or 
awkward to directly monitor the position and attitude of the head with respect to the 
body. Thus it will be understood that the translations and transformations of FIG. 2, while 
instructive in teaching how to carry out the invention by the use of an illustrative series of 
related coordinate systems, such teachings may be subject to modification to serve 

1 5 different monitoring and signal processing embodiments of the invention. 

FIG. 2 further illustrates another coordinate system (x', y* , z') having an origin 67 fixed in 
an eye 68 of the viewer's head 50. For stereoscopic embodiments of the present 
invention, two such eye coordinate systems may be used, although only one is shown. 
20 The X, y, z head coordinate system is used as a reference for measurements of the attitude 
of the viewer's head. The x', y', z' eye coordinate system is used as a reference for 
measurements of the attitude of the viewer's eye. 

The eyes 12, 14 of FIG. 1 may view objects in the object space 44 of FIG. 2 by means of 
25 a head-mounted display (not shown) mounted on the head 16 or 50. The display may be 
setup so that the space within which the viewer is located is not visible but only sensed 
visually by means of images. Or, the space may be made visible, with parts represented 
by images of virtual objects that may be infraposed, interposed or superimposed. 

30 In order to properly position the object space's coordinate system 44 with respect to 

viewer's head coordinate system, as utilized by a head mounted display, according to the 
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present invention, it is useful to conceive of the four separate coordinate systems having 
the separate origins 56, 52, 67 and reference frames freely translating and freely rotating 
with respect to each other and the origin 44 and its reference frame. In fact, the origins 
52, 67 will be approximately fixed with regard to translations but viewing them as freely 
5 translatable does not unduly complicate the mathematical transformations and translation 
of coordinates. Such a translation can be omitted, however, in most applications. As 
pointed out earlier, the translational position and the attitude of the head can be measured 
directly with respect to the object space and the body's position and orientation ignored, if 
desired. Such is within the scope of the present invention. 

10 

With regard to translation, as known in the art of analytic geometry, two coordinate 
systems having their origins translating out of coincidence can be brought into 
coincidence by means of a parallel shift. 

15 I.e., if the origin 46 of the object space has coordinates , a^^ with respect to the 
origin 56 of the coordinate system in the body 54, then the relations 

x*=x**-l-a, 
20 y*=y**+a2 

hold between the coordinates x*, y*, z* of a point 70 of space with respect to the body 54 
25 of the viewer and the coordinates x**, y**, z** of the same point 70 with respect to the 
object space 44. If the body is in motion and its translatory position is monitored then a, , 
and will be changing according to the monitored position of the body with respect 
to the inertial reference system. 

30 Similarly, as is also known, with regard to rotation, two systems having the same origin, 
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or having their origins brought into coincidence by the above transformation, but having 
their axes nonahgned, can be brought into alignment using direction cosines or using 
Euler angles or similar techniques which are or may be equally valid approaches. 



5 In the case of direction cosines, each axis of one system is thought of as making an angle 
with each axis of the other system. The cosines of these angles are denoted by a.^. , where 

i and k run through the values 1, 2 and 3. In the following example, the first index refers 
to the X*, y*, z* system and the second index to the x**, y**, z** system. The index 1 
corresponds to the x*- or x**-axis, 2 to the y*- or y**-axis and 3 to the z*- or z**-axis; 
10 that is. 



15 



25 



a,, = cos( jc* , jc** ) a,2 =cos(jc*, y**) a^^ =cos(x', z") 
=cos(3;\^") =cos(>'*, a^^ =cos{y\ z") 

z , x ) = cos( z , 3; ) «33 = cos( z , z ) 

where the arguments refer to the angles in the planes formed by the specified axes. 
20 The coordinates of an arbitrary point then transform according to the following equations: 
X* = flEjjX** + a,2y** + a,3Z** 

y* = ^21^** '^22y** a^^z** 
z* = <33, X** + <232y** + a33Z**. 

The a-f^ are called "direction cosines." The Euler angle or the Euler theorem approach 
would be similar and will not be described in detail as it will be evident to one skilled in 
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the art of analytic geometry as to how to proceed. Similarly, other methods of 
transformation are known, including more general methods, and by describing one such 
method it is certainly not intended to exclude others. 



5 For the special case of the present invention, the body and object space coordinate 

systems may be viewed as being both translated and rotated with respect to each other at 
the same time. This case is a combination of the two cases considered above and leads to 
the following equations of transformation: 

10 x*=ai +aii x**-i-ai2 y**+ai3 z** 

y*=a2 +a2i x**+a22 y**+a23 z** 

z*=a3 +a3i x**+a32 y**+a33 z**. 

15 

The image control 40 of FIG. 1 may be used to concentrate the imaging power of an 
image processor (not shown) in a relatively small area that tracks the viewer's visual axis. 
To do this, it is necessary to make two additional sets of transformations. I.e., if point 52 
is the origin of the head and it is desired to further transform to head coordinates the same 
20 point 70 transformed above from object space to body coordinates. If the body's origin is 
translated to that of the head by sensed distances bi, b2, bs, and the head coordinate 
system is rotated with respect to the body coordinate system as defined by nine direction 
cosines bik (defined in a manner similar to that in which the aik direction cosines were 
defined), then the coordinates of the same point 70 in head coordinates are: 

25 

x=bi +bii x*-i-bi2 y*+bi3 z* 
y=b2 +b2i x*-i-b22 y*+b23 z* 

30 Z=b3 +b3l X*+b32 y*+b33 z*. 
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and only one more transformation is required, i.e., from head to eye coordinates. This is 
done by the use of nine direction cosines Cjk, similarly used as follows: 



x'=Ci +Cii X+C12 y+ci3 z 

5 

y'=C2 +C21 X+C22 y+C23 z 
Z "C3 +C31 X+C32 y+C33 z, 

10 and the designer is then able to provide an image artifact on, in, or vmder an image, a 
highly detailed image in a small area, a greater dynamic image range in a small area of 
the overall image, or various combinations thereof, according to the present invention. In 
the last mentioned equations above, the eye may be assumed to be fixed in translatory 
position with respect to the head so that ci, C2, C3 are constants. It should be realized that 

15 the order of transformations of coordinate systems described above may be carried out in 
any order or even without any particular order. The same may be said for translations. 
And if it is desired to omit a coordinate system or a degree of freedom in a given system, 
such may be done as well. For instance, it may be deemed acceptable to track only two 
degrees of freedom of an eye, e.g., ductions only, omitting torsions. It is even conceivable 

20 that tracking of only one degree of freedom is desired, such as horizontal ductions only. 
As another example, the position of the head may be assumed to be fixed with respect to 
the body. In that case, bi, b2, bs in the above mentioned equations will be constants 
instead of being monitored translational positions. Similarly, the head or body coordinate 
systems may even be omitted, for example. 

25 

Points in the object space coordinate system 44 expressed in head coordinates may be 
projected or transformed from the three dimensions of the object space to the two 
dimensions of the display 28 screen, i.e., a decrease in dimensionality (a dimensionality 
reduction is not a requirement or limitation, since a projection, for example onto a curved 
30 surface might be needed for some applications). This can be thought of as a shadow 

projection except being a contracted "shadow" rather than the expanded type of everyday 
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experience. 



For example, as shown by an edge-on view of a screen 72 in FIG. 3, and not by way of 
limitation, we can consider an origin 74 of the screen coordinate system for the illustrated 
5 eye 68 to be in a defined center of the screen 72 with the screen xs - and ys -axes in the 
plane of the screen parallel to the respective x'- and y*-axes of the eye coordinate system, 
for example. In that case, the eye z*-axis perpendicularly intersects the screen at its origin 
74. The eye's point of view 67 lies on this axis at a distance D behind the screen at a point 
which may be translated with respect to the origin 52 of the head coordinate system. 

10 

Now, consider a point 76 with eye coordinates x\y\z\ (These coordinates may have been 
generated from object space coordinates using the transformations previously described). 
FIG. 3 represents the components of this point in the eye system's x -z'-plane. Applying 
the well-known laws of similar triangles, it can be seen that if xs designates the x- 
1 5 component of the point in screen coordinates, 

xsD=XhVzh, 

or, solving for xs, 

20 

Xs =D(xh 7zh 

Similarly, in the eye y -z*-plane (not shown), 
25 ys =D(yh Vzh '), 

where ys is the y-component of the point in screen coordinates. As in all of the other 
coordinate transformations described previously, there are other methods of projection 
and corresponding methods for accomplishing such transformations. In this case, a 
30 particular transformation from three-dimensional space to two-dimensional space is 
illustrated, but it is not by any means intended to exclude such other transformations, 
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projections or methods. 

A refinement to the above illustrated approach is to modify the value of D for points near 
the edges of the screen, to maintain a constant or approximately constant relationship 
5 between the linear separation between the two points, in screen coordinates, and their 
angular separation at the viewer's eye. This may be desirable when the angles subtended 
at the eye by the screen edges are large. 

One may desire to express the screen coordinates in a coordinate system having its origin 
10 in the top left comer of the screen, as is usual in the art of computer graphics. This may 
be effected by a simple translation between the screen coordinate system described above 
and the comer-originated screen system. 

FIG. 4 shows a method for carrying out the present invention. After entering in a step 79, 

15 eye attitude, as shown in FIG. 2, is sensed by the one or more eye monitors 20, 22 of FIG. 
1 in a step 80 and the signal 30 or the signal 32 or both of FIG. 1 are provided to the 
computer 36. Head 16 attitude, as shown in connection with FIG. 2, is sensed with 
respect to the body 54 or the object space 44 in a step 81 and the signal on the line 28 is 
also provided to the computer 36. A step 82 is next illustrated to show that in some 

20 application it may be desired to sense head position. If so, a step 83 is executed to sense 
head position. If not, a step 84 is executed to sense body position. In either event, a step 
85 is next executed to determine if it is desired to sense body attitude. It should be 
realized that the step 82 need not be actually present in any particular embodiment of the 
invention, since it will already be known in advance by the designers whether it is desired 

25 to sense head position or not. Thus, the desired sensing steps would be executed directly 
without need for execution of a decision step, such as step 82. It will therefore be realized 
that the decision block 82 is merely illustrative of various design courses that might be 
selected for different embodiments of the invention. This may be said for steps 85, 87, 88 
below, as well. If it is desired to sense body attitude, it is sensed in a step 86. A body 

30 center such as a selected point 89 in the body 18 of FIG. 1 or an origin 56 in a body 54 in 
FIG. 2, e.g., in the trunk area just below the head 16 or 50, respectively, may be used a 
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body reference point for this purpose. If not, a decision step 87 is illustrative of whether 
or not body position was already sensed in the step 84. If not, a step 88 is executed to 
determine if it is desired to sense body position. If so, a step 89 is executed to do so. If 
not, or if the illustrative step 87 indicates that body position has already been sensed, and 
5 a step 90 is executed to compute the origin and direction of at least one visual vector of 
the eyes 12, 14 of FIG. 1 with respect to a selected reference frame such as the reference 
space 44 of FIG. 2. Once this is computed, a step 91 is executed to provide a visual axis 
signal on the line 38 of FIG. 1. The control 40 uses the signal on the line 38 to provide 
the control signal on the line 42 (which may be an image control signal) as indicated by a 
10 step 92 of FIG. 4. A return is then made in a step 93. 

The present invention may be used for a variety of purposes. For example, as shown in 
FIG. 5, a display 1 14 in an apparatus 10a can be made to operate in a manner suggested 
above in connection with the controls of FIG. 1. Controls similar to those of FIG. 1 are 

15 located in an object space 138 of FIG. 5. Nonuniform resolution images, images having 
nonuniform dynamic range, or the like, are provided on the display 1 14 in an image space 
1 15 to a passive viewer 116 having a left eye 117 and a right eye 118. The images may be 
provided stereoscopically from the single image source 114 by providing alternate left 
and right eye images on lines 119, 120 to left and right light valves 121, 122. The visual 

20 axes of the passive viewer's eyes will naturally follow those small portions of the 
respective left and right images with more image information concentrated therein. 

A control 123 provides image signals on a line 124 and a control signal on a Une 125 to 
the display 1 14. For stereoscopic embodiments the control 123 may be used to provide 

25 control signals on lines 126, 127 for controlling the light valves 121, 122, respectively. 
The control 123 is responsive to an incoming encoded image signal on a line 128. The 
signal on the line 128 may be provided by a receiver 129 that is connected to an antenna 
signal line 130 responsive to a transmitted space signal 132 transmitted by a transmitting 
antenna 136 in an object space 138 and picked up by an antenna 134. Of course, the 

30 signal need not be broadcast but may be provided in any known way such as by video 
cassette, cable, optical fiber, satellite, or the like. 
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The signal on the hne 130 may be created in the object space 138 by a cameraman 140 
using one or more cameras such as a pair of cameras 142, 144 mounted on either side of 
the cameraman's head for picking up images of objects in the object space 138 such as an 
5 object 146 which provides reflected light on lines 148 from a point 150 gazed upon by 
the cameraman 140 by a conjunction of respective visual axes 152, 154 of left and right 
eyes 156, 158 of the cameraman 140. The eyes 156, 158 are monitored by respective eye 
position monitors 1 60, 1 62 which may be oculometers that send out and receive back 
infrared signals on lines 164, 166. As mentioned, there are of course other ways to 

10 monitor eyes besides oculometers. Sensed eye position signals are provided on lines 168, 
170 to controls 172, 174 which play the role of the signal processor 36 of FIG. 1. It 
should be realized that the monitors 160, 162 will be similar to the eye attitude monitors 
20, 22 of FIG. 1 and that the signals on the lines 168, 170 may include fiiUy three axis 
eye attitude information. The controls 172, 174 provide camera control signals on lines 

15 176, 178 which are similar to the signal on the line 38 of FIG. 1. In response, the cameras 
142, 144 provide image signals on lines 180, 182 to the controls 172, 174 which provide 
a pair of encoded signals on lines 176, 178, which may be similar to the signal on the line 
42 of FIG. 1 except in stereo as in the example given. These may be alternately switched 
(time division multiplexed) by a switch 184 of FIG. 5 onto a single signal line 186 as 

20 controlled by a signal on a line 188. A transmitter 190 may be used to boost the signal on 
the line 186 to provide a boosted signal on a line 192 to the antenna 136. As mentioned, 
broadcast is illustrated but any other method of image delivery after such image 
acquisition may be utilized. 

25 The object space may include a plurality of microphones 200, 202, 204 arranged aroimd 
the cameraman's head for providing a corresponding plurality of sensed sound signals on 
lines 206, 208, 210. One or both of the controls 172, 174 encodes the information in these 
sensed signals onto one or both of the signals onto the lines 176, 178 for use in speakers 
214, 216, 218 in the image space 101 as provided by decoded signals on lines 220, 222, 

30 224 by the control 116. 
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Additionally, as suggested in FIG. 1, the head 16 and even the body 18 of the cameraman 
140 may be monitored in order to transform and/or translate eye coordinates into one or 
more related coordinate systems as suggested in FIG. 2. The viewer is then able to 
experience images, albeit passively, from the perspective of the cameraman as he moves 
5 about in the object space as if the viewer were the cameraman himself. 

Instead of the single display 114 providing the separate halves of each stereopair 
altemately in succession, a pair of separate displays 250, 252 may be provided as shown 
in an image space B 254. Components shown in image space B are similar to those 

10 shown in image space A 1 15 and are similarly labeled. It should be realized that the 
images of the present invention need not be displayed stereoscopically but may be 
presented from a single point of view as well. The images may be provided as shown in 
approaches shown by U.S. Pat. Nos. 4,5 15,450 or 4,427,274 or PCT Patent WO 86/01310 
in conjunction with, e.g., a pair of light shutter or polarizer glasses (not shown) such as 

1 5 shown in U.S. Pat. No. 4,424,529, or may be provided via image sources in a helmet for 
moimtrng on a viewer's head in an approach suggested by U.S. Pat. Nos. 4,636,866; 
4,968,123; 4,961,626; 4,969,714; 4,310,849; the NASA 3-D Hehnet (Electronic 
Engineering Times- Jan. 13, 1986, pp.1 & 22); the Sony Visortron (Time, Dec. 28, 1992, 
p.l 1; Popular Science, March, 1993, p.26), or many other possible presentation 

20 approaches. 

FIG. 6 shows another application of the present invention, whereby a viewer 300 in an 
image space 302 having a display 304 for presenting successive images to the .viewer's 
eyes 306, 308. The display 304 may always provide successive images from the same 

25 perspective, i.e., nonstereoscopically, or may altemately provide the separate halves of 
stereopair images. Or, stereopairs may be provided by separate displays 310, 312 for 
separate halves of the stereopairs, one half for each eye. The display may be provided 
such as described in U.S. Pat Nos. 4,515,450 or 4,427,274 PCT Patent WO 86/01310 in 
conjunction with, e.g., a pair of light shutter or polarizer glasses (not shown) such as 

30 shown in U.S. Pat. No. 4,424,529, or may be provided via image sources in a helmet for 
mounting on a viewer's head in an approach suggested by U.S. Pat. Nos. 4,636,866; 
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4,968,123; 4,961,626; 4,969,714; 4,310,849; the NASA 3-D Helmet (Electronic 
Engineering Times- Jan. 13, 1986, pp.1 & 22); the Sony Visortron (Time, Dec. 28, 1992, 
p.l 1; Popular Science, March, 1993, p.26), or many other possible presentation 
approaches. 

5 

A decoder 314 is responsive to an encoded image signal on a line 316 for providing a 
display signal on a line 318 to the display 304. The encoded image signal on the line 316 
may be provided by an image source 320 which may be an image store containing a very 
large plurality of selectable stored images such as may be consistent with "virtual reality" 

10 and which may be selected according to a selection signal on a line 321 that represents 
the visual axes or vectors of the eyes 306, 308 in the space 302. A viewer body part 
monitor signal on a line 322 from a viewer body part monitor 324 represents one or more 
monitors such as suggested in FIG. 1 is combined with the sensed eye attitude signals as 
suggested in connection with FIG. 2. Such a head part monitor may be provided as shown 

15 in U.S. Pat. Nos. 4,988,981; 5,097,252; 4,937,444; 4,542,291; or Polhemus Navigation 
Sciences U.S. Pat. Nos. 3,983,474 and 4,017,858 and like patents which are hereby 
incorporated in their entirety by reference. If a Polhemus monitor would not be 
applicable, other head mounted sensing devices such as inertial sensors could be used 
including accelerometers, any of the known type of g)TOS, or the like. The body part 

20 monitor 324 may sense motion of a selected part of the body of the viewer, such as a 
head or hand, or both, or arm, trunk or leg, as indicated by a sensing line 326 which, in 
the case of multiple body part monitoring, represents more than one signal. For example, 
the position of the body or head of the viewer 300 in the image space may be monitored 
and the attitude (pitch, roll and yaw) of the viewer's head with respect to the body or the 

25 object space may also be monitored as suggested previously. 

A variable magnification device 328 may be situated in between the viewer 300 and the 
display 304 and is responsive to a control signal on a line 330 for providing images from 
the display 304 to the viewer 300 at various apparent distances. (A similar variable 
30 magnification device may be provided for the passive viewer of FIG. 5 except the degree 
of magnification will then be under the control of the cameraman's eyes. The device 328 
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of FIG. 6 may be a unitary device or may comprise separate devices 328a, 328b, one 
situated before each of the viewer's eyes 306, 308, respectively. A computer 340 (similar 
to the signal processor 36 of FIG. 1) is responsive to viewer eye monitor signals on lines 
342, 344 and to the body part monitor signal(s) on the line 322 for providing the control 
5 signals on the lines 321, 330. The eye monitor signals on the lines 342, 344 are provided, 
respectively, by left and right monitors 346, 348 which may be oculometer devices such 
as invented by John Merchant of Honeywell. Such an oculometer is disclosed in U.S. Pat. 
No. 3,462,604. The left and right eye monitors 346,348 are responsive to left and right 
eye 306,308 movements, respectively. Numerous eye tracking devices, other than 

10 oculometers, are generally known in the art of eye tracking. An example of another type 
of eye tracker, based on the detection of Purkinje images, is disclosed in U.S. Pat. No. 
3,712,716. Still another example of a type of eye tracker is disclosed in U.S. Pat. No. 
4,561,448, based on electro-oculography. The abovementioned eye monitoring patents 
are hereby incorporated by reference in their entirety. These are examples only and 

1 5 should not be taken as limiting the choice of eye trackers or eye tracking methods, as any 
type of eye tracking method or apparatus capable of tracking the position of the visual 
axis of the cameraman's eyes 156, 158 or the viewer's eyes 306, 308 as encompassed by 
the monitors 160, 162 or 346, 348 as shown in FIGS. 5 & 6, respectively. The 
embodiment of the invention shown in FIG. 6 is of course for an active viewer while FIG. 

20 5 shows a passive viewer. 

Although the invention has been shown and described with respect to a best mode 
embodiment thereof, it should be understood by those skilled in the art that the foregoing 
and various other changes, omissions, and additions in the form and detail thereof may be 
25 made therein without departing from the spirit and scope of the invention. 
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